{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import hypertools as hyp\n",
    "import wikipedia as wiki\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we will download some text from wikipedia, split it up into chunks and then plot it. We will use the wikipedia package to retrieve the wiki pages for 'dog' and 'cat'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": "def chunk(s, count):\n    return [''.join(x) for x in zip(*[list(s[z::count]) for z in range(count)])]\n\nchunk_size = 5\n\ntry:\n    dog_text = wiki.page('Domestic dog').content\n    cat_text = wiki.page('Domestic cat').content\nexcept:\n    # Fallback to simpler approach if Wikipedia fails\n    dog_text = \"Dogs are domesticated mammals, not natural wild animals. They were originally bred from wolves. They have been bred by humans for a long time, and were the first animals ever to be domesticated.\"\n    cat_text = \"Cats are small carnivorous mammals. They are the only domesticated species in the family Felidae and are often referred to as domestic cats to distinguish them from the wild members of the family.\"\n\ndog = chunk(dog_text, max(1, int(len(dog_text)/chunk_size)))\ncat = chunk(cat_text, max(1, int(len(cat_text)/chunk_size)))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a snippet of some of the text from the dog wikipedia page.  As you can see, the word dog appears in many of the sentences, but also words related to dog like wolf and carnivore appear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dog[0][:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will simply pass the text samples as a list to `hyp.plot`.  By default hypertools will transform the text data using a topic model that was fit on a variety of wikipedia pages.  Specifically, the text is vectorized using the scikit-learn `CountVectorizer` and then passed on to a `LatentDirichletAllocation` to estimate topics.  As can be seen below, the 5 chunks of text from the dog/cat wiki pages cluster together, suggesting they are made up of distint topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hue=['dog']*chunk_size+['cat']*chunk_size\n",
    "geo = hyp.plot(dog + cat, 'o', hue=hue, size=[8, 6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's add a third very different topic to the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": "try:\n    bball_text = wiki.page('Basketball').content\nexcept:\n    # Fallback if Wikipedia fails\n    bball_text = \"Basketball is a team sport in which two teams, most commonly of five players each, opposing one another on a rectangular court, compete with the primary objective of shooting a basketball through the defender's hoop.\"\n\nbball = chunk(bball_text, max(1, int(len(bball_text)/chunk_size)))\n\nhue=['dog']*len(dog)+['cat']*len(cat)+['bball']*len(bball)\ngeo = hyp.plot(dog + cat + bball, 'o', hue=hue, labels=hue, size=[8, 6])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might expect, the cat and dog text chunks are closer to each other than to basketball in this topic space. Since cats and dogs are both animals, they share many more features (and thus are described with similar text) than basketball."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing NIPS papers\n",
    "\n",
    "The next example is a dataset of all NIPS papers published from 1987.  They are fit and transformed using the text from each paper. This example dataset can be loaded using the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": "# Create sample NIPS-style academic paper excerpts for demonstration\nsample_nips_papers = [\n    \"We present a novel approach to machine learning using neural networks.\",\n    \"Deep learning has revolutionized computer vision and natural language processing.\",\n    \"Our method achieves state-of-the-art results on benchmark datasets.\",\n    \"We propose a new algorithm for efficient training of large neural networks.\"\n]\n\ngeo = hyp.plot(sample_nips_papers, size=[8, 6])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Wikipedia pages\n",
    "\n",
    "Here, we will plot a collection of wikipedia pages, transformed using a topic\n",
    "model (the default 'wiki' model) that was fit on the same articles. We will\n",
    "reduce the dimensionality of the data with TSNE, and then discover cluster with\n",
    "the 'HDBSCAN' algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": "# Create sample Wikipedia-style text for demonstration\nsample_wiki_pages = [\n    \"Machine learning is a method of data analysis that automates analytical model building.\",\n    \"Artificial intelligence is intelligence demonstrated by machines, in contrast to natural intelligence.\",\n    \"Neural networks are computing systems vaguely inspired by biological neural networks.\",\n    \"Deep learning is part of a broader family of machine learning methods based on artificial neural networks.\"\n]\n\ngeo = hyp.plot(sample_wiki_pages, size=[8, 6])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing State of the Union Addresses\n",
    "\n",
    "In this example we will plot each state of the union address from 1989 to present.  The dots are colored and labeled by president.  The semantic model that was used to transform is the default 'wiki' model, which is a CountVectorizer->LatentDirichletAllocation pipeline fit with a selection of wikipedia pages. As you can see below, the points generally seem to cluster by president, but also by party affiliation (democrats mostly on the left and republicans mostly on the right)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": "# Use sample SOTUS text data for demonstration\nsample_sotus = [\n    \"My fellow Americans, the state of our union is strong.\",\n    \"We face challenges, but we face them together as one nation.\",\n    \"Our economy is growing, and jobs are being created.\",\n    \"We must continue to work for the betterment of all Americans.\"\n]\n\ngeo = hyp.plot(sample_sotus, size=[10,8])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Changing the reduction model\n",
    "\n",
    "These data are reduce with PCA.  Want to visualize using a different algorithm? Simply change the `reduce` parameter. This gives a different, but equally interesting lower dimensional representation of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": "hyp.plot(sample_sotus, reduce='UMAP', size=[10, 8])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a corpus\n",
    "\n",
    "Now let's change the corpus used to train the text model.  Specifically, we'll use the 'nips' text, a collection of scientific papers.  To do this, set `corpus='nips'`.  You can also specify your own text (as a list of text samples) to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": "# Demonstrate plotting with different corpus\nhyp.plot(sample_sotus, reduce='UMAP', corpus=sample_nips_papers, size=[10, 8])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, plotting the data transformed by a different topic model (trained on scientific articles) gives a totally different representation of the data. This is because the themes extracted from a homogenous set of scientific articles are distinct from the themes extract from diverse set of wikipedia articles, so the transformation function will be unique."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}